ONNX
wangqiang9 commited on
Commit
14df15c
·
verified ·
1 Parent(s): 774970e

Upload folder using huggingface_hub

Browse files
Files changed (9) hide show
  1. .msc +0 -0
  2. .mv +1 -0
  3. README.md +45 -48
  4. configuration.json +1 -0
  5. face_det.onnx +3 -0
  6. face_landmark.onnx +3 -0
  7. fantasyportrait_model.ckpt +3 -0
  8. pd_fpg.pth +3 -0
  9. upload.py +12 -0
.msc ADDED
Binary file (843 Bytes). View file
 
.mv ADDED
@@ -0,0 +1 @@
 
 
1
+ Revision:master,CreatedAt:1754971377
README.md CHANGED
@@ -1,103 +1,100 @@
1
- ---
2
- license: apache-2.0
3
- datasets:
4
- - acvlab/FantasyPortrait-Multi-Expr
5
- language:
6
- - en
7
- ---
8
- # FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers
 
9
 
10
  [![Home Page](https://img.shields.io/badge/Project-FantasyPortrait-blue.svg)](https://fantasy-amap.github.io/fantasy-portrait/)
11
  [![arXiv](https://img.shields.io/badge/Arxiv-2507.12956-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2507.12956)
12
- [![hf_dataset](https://img.shields.io/badge/🤗%20Dataset-FantasyPortrait-yellow.svg)](https://huggingface.co/datasets/acvlab/FantasyPortrait-Multi-Expr)
13
  [![hf_paper](https://img.shields.io/badge/🤗-FantasyPortrait-red.svg)](https://huggingface.co/papers/2507.12956)
14
 
15
- ## 🔥 Latest News!!
16
- * August 10, 2025: We released the inference code, model weights and datasets.
17
 
18
- ## Demo
19
- For more interesting results, please visit our [website](https://fantasy-amap.github.io/fantasy-portrait/).
20
 
21
  | ![单人示例](./danren_1.gif) | ![对比](./duibi.gif) |
22
  | :---: | :---: |
23
  | ![动物](./dongwu.gif) | ![双人1](./shuangren_1.gif) |
24
  | ![双人2](./shuangren_2.gif) | ![三人](./sanren.gif) |
25
 
 
 
26
 
27
- ## Quickstart
28
- ### 🛠️Installation
29
-
30
- Clone the repo:
31
 
32
  ```
33
  git clone https://github.com/Fantasy-AMAP/fantasy-portrait.git
34
  cd fantasy-portrait
35
  ```
36
 
37
- Install dependencies:
38
  ```
39
  apt-get install ffmpeg
40
- # Ensure torch >= 2.0.0
41
  pip install -r requirements.txt
42
- # Note: flash attention must be installed
43
  pip install flash_attn
44
  ```
45
 
46
- ### 📦Multi-Expr Dataset
47
- We make public the first multi-portrait facial expression video dataset **Multi-Expr Dataset**, Please download it via the this [link](https://huggingface.co/datasets/acvlab/FantasyPortrait-Multi-Expr).
48
 
49
- ### 🧱Model Download
50
- | Models | Download Link | Notes |
 
51
  | --------------|-------------------------------------------------------------------------------|-------------------------------|
52
- | Wan2.1-I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | Base model
53
- | FantasyPortrait | 🤗 [Huggingface](https://huggingface.co/acvlab/FantasyPortrait/) 🤖 [ModelScope](https://www.modelscope.cn/models/amap_cvlab/FantasyPortrait/) | Our emo condition weights
54
 
55
- Download models using huggingface-cli:
56
  ``` sh
57
  pip install "huggingface_hub[cli]"
58
  huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
59
  huggingface-cli download acvlab/FantasyPortrait --local-dir ./models
60
  ```
61
 
62
- Download models using modelscope-cli:
63
  ``` sh
64
  pip install modelscope
65
  modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
66
  modelscope download amap_cvlab/FantasyPortrait --local_dir ./models
67
  ```
68
 
69
- ### 🔑 Single-Portrait Inference
70
  ``` sh
71
  bash infer_single.sh
72
  ```
73
 
74
- ### 🔑 Multi-Portrait Inference
75
- If you use input image and drive videos with multiple people, you can run as follows:
76
  ``` sh
77
  bash infer_multi.sh
78
  ```
79
- If you use input image with multiple people and different multiple single-human driven videos, you can run as follows:
80
- ```sh
81
- bash infer_multi_diff.sh
82
- ```
83
 
84
- ### 📦Speed and VRAM Usage
85
- We present a detailed table here. The model is tested on a single A100.
86
 
87
- |`torch_dtype`|`num_persistent_param_in_dit`|Speed|Required VRAM|
 
 
 
88
  |-|-|-|-|
89
- |torch.bfloat16|None (unlimited)|15.5s/it|40G|
90
- |torch.bfloat16|7*10**9 (7B)|32.8s/it|20G|
91
- |torch.bfloat16|0|42.6s/it|5G|
92
 
93
 
94
 
95
- ## 🧩 Community Works
96
- We ❤️ contributions from the open-source community! If your work has improved FantasyPortrait, please inform us.
97
- Or you can directly e-mail [[email protected]](mailto://[email protected]). We are happy to reference your project for everyone's convenience.
98
 
99
- ## 🔗Citation
100
- If you find this repository useful, please consider giving a star ⭐ and citation
101
  ```
102
  @article{wang2025fantasyportrait,
103
  title={FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers},
@@ -107,5 +104,5 @@ If you find this repository useful, please consider giving a star ⭐ and citati
107
  }
108
  ```
109
 
110
- ## Acknowledgments
111
- Thanks to [Wan2.1](https://github.com/Wan-Video/Wan2.1), [PD-FGC](https://github.com/Dorniwang/PD-FGC-inference) and [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) for open-sourcing their models and code, which provided valuable references and support for this project. Their contributions to the open-source community are truly appreciated.
 
1
+ ---
2
+ frameworks:
3
+ - Pytorch
4
+ license: Apache License 2.0
5
+ tasks:
6
+ - text-to-video-synthesis
7
+
8
+
9
+ # FantasyPortrait:基于表情增强扩散变换器的多角色肖像动画生成
10
 
11
  [![Home Page](https://img.shields.io/badge/Project-FantasyPortrait-blue.svg)](https://fantasy-amap.github.io/fantasy-portrait/)
12
  [![arXiv](https://img.shields.io/badge/Arxiv-2507.12956-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2507.12956)
13
+ [![hf_dataset](https://img.shields.io/badge/🤗%20Dataset-FantasyPortrait-yellow.svg)](https://huggingface.co/datasets/acvlab/FantasyPortrait)
14
  [![hf_paper](https://img.shields.io/badge/🤗-FantasyPortrait-red.svg)](https://huggingface.co/papers/2507.12956)
15
 
16
+ ## 🔥 最新动态!!
17
+ * 2025年8月10日:我们已发布推理代码、模型权重和数据集。
18
 
19
+ ## 演示
20
+ 更多有趣的结果,请访问我们的[网站](https://fantasy-amap.github.io/fantasy-portrait/)
21
 
22
  | ![单人示例](./danren_1.gif) | ![对比](./duibi.gif) |
23
  | :---: | :---: |
24
  | ![动物](./dongwu.gif) | ![双人1](./shuangren_1.gif) |
25
  | ![双人2](./shuangren_2.gif) | ![三人](./sanren.gif) |
26
 
27
+ ## 快速开始
28
+ ### 🛠️ 安装
29
 
30
+ 克隆仓库:
 
 
 
31
 
32
  ```
33
  git clone https://github.com/Fantasy-AMAP/fantasy-portrait.git
34
  cd fantasy-portrait
35
  ```
36
 
37
+ 安装依赖:
38
  ```
39
  apt-get install ffmpeg
40
+ # 确保 torch >= 2.0.0
41
  pip install -r requirements.txt
42
+ # 注意:必须安装 flash attention
43
  pip install flash_attn
44
  ```
45
 
46
+ ### 📦 Multi-Expr 数据集
47
+ 我们公开了首个多人肖像面部表情视频数据集 **Multi-Expr Dataset**,请通过以下链接下载:
48
 
49
+
50
+ ### 🧱 模型下载
51
+ | 模型 | 下载链接 | 说明 |
52
  | --------------|-------------------------------------------------------------------------------|-------------------------------|
53
+ | Wan2.1-I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | 基础模型
54
+ | FantasyPortrait | 🤗 [Huggingface](https://huggingface.co/acvlab/FantasyPortrait/) 🤖 [ModelScope](https://www.modelscope.cn/models/amap_cvlab/FantasyPortrait/) | 我们的表情条件权重
55
 
56
+ 使用 huggingface-cli 下载模型:
57
  ``` sh
58
  pip install "huggingface_hub[cli]"
59
  huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
60
  huggingface-cli download acvlab/FantasyPortrait --local-dir ./models
61
  ```
62
 
63
+ 使用 modelscope-cli 下载模型:
64
  ``` sh
65
  pip install modelscope
66
  modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
67
  modelscope download amap_cvlab/FantasyPortrait --local_dir ./models
68
  ```
69
 
70
+ ### 🔑 单人肖像推理
71
  ``` sh
72
  bash infer_single.sh
73
  ```
74
 
75
+ ### 🔑 多人肖像推理
 
76
  ``` sh
77
  bash infer_multi.sh
78
  ```
 
 
 
 
79
 
 
 
80
 
81
+ ### 📦 速度与显存占用
82
+ 我们在��提供详细表格。模型在单张A100上进行测试。
83
+
84
+ |`torch_dtype`|`num_persistent_param_in_dit`|速度|所需显存|
85
  |-|-|-|-|
86
+ |torch.bfloat16|None (无限制)|15.5秒/迭代|40G|
87
+ |torch.bfloat16|7*10**9 (7B)|32.8秒/迭代|20G|
88
+ |torch.bfloat16|0|42.6秒/迭代|5G|
89
 
90
 
91
 
92
+ ## 🧩 社区贡献
93
+ 我们 ❤️ 来自开源社区的贡献!如果您的工作改进了 FantasyPortrait,请告知我们。
94
+ 您也可以直接发送邮件至 [[email protected]](mailto://[email protected])。我们很乐意引用您的项目,方便大家使用。
95
 
96
+ ## 🔗 引用
97
+ 如果本仓库对您有帮助,请考虑给我们一个 star ⭐ 并引用以下论文:
98
  ```
99
  @article{wang2025fantasyportrait,
100
  title={FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers},
 
104
  }
105
  ```
106
 
107
+ ## 致谢
108
+ 感谢 [Wan2.1](https://github.com/Wan-Video/Wan2.1)[PD-FGC](https://github.com/Dorniwang/PD-FGC-inference) [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 开源他们的模型和代码,为本项目提供了宝贵的参考和支持。我们非常感谢他们对开源社区的贡献。
configuration.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"framework":"Pytorch","task":"text-to-video-synthesis"}
face_det.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ea8de1da304c1459a11f637798bb1140805365aeb3cf6637ca6d61909720aec
3
+ size 370619
face_landmark.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:463aaa48dd24b9d3b966c76accf9a27f7c590a73a3827d5002a975e46bd54995
3
+ size 2865751
fantasyportrait_model.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95e1b17849ba13766ea21bd9a9a160b115e80f5dc9adf810da600a9b606c81df
3
+ size 4013364090
pd_fpg.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b34f69d8e6070d1f1e9986a31bc954a8998d45ae74436aa7de128825b3606007
3
+ size 39542043
upload.py ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from huggingface_hub import HfApi
2
+
3
+ api = HfApi()
4
+ local_dir = "/Users/zuojing/wangqiang/FantasyPortrait"
5
+ repo_id = "acvlab/FantasyPortrait"
6
+
7
+ # 上传文件夹
8
+ api.upload_folder(
9
+ folder_path=local_dir,
10
+ repo_id=repo_id,
11
+ repo_type="model"
12
+ )