Upload folder using huggingface_hub
Browse files- .msc +0 -0
- .mv +1 -0
- README.md +45 -48
- configuration.json +1 -0
- face_det.onnx +3 -0
- face_landmark.onnx +3 -0
- fantasyportrait_model.ckpt +3 -0
- pd_fpg.pth +3 -0
- upload.py +12 -0
.msc
ADDED
Binary file (843 Bytes). View file
|
|
.mv
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
Revision:master,CreatedAt:1754971377
|
README.md
CHANGED
@@ -1,103 +1,100 @@
|
|
1 |
-
---
|
2 |
-
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
-
|
7 |
-
|
8 |
-
|
|
|
9 |
|
10 |
[](https://fantasy-amap.github.io/fantasy-portrait/)
|
11 |
[](https://arxiv.org/abs/2507.12956)
|
12 |
-
[](https://huggingface.co/datasets/acvlab/FantasyPortrait
|
13 |
[](https://huggingface.co/papers/2507.12956)
|
14 |
|
15 |
-
## 🔥
|
16 |
-
*
|
17 |
|
18 |
-
##
|
19 |
-
|
20 |
|
21 |
|  |  |
|
22 |
| :---: | :---: |
|
23 |
|  |  |
|
24 |
|  |  |
|
25 |
|
|
|
|
|
26 |
|
27 |
-
|
28 |
-
### 🛠️Installation
|
29 |
-
|
30 |
-
Clone the repo:
|
31 |
|
32 |
```
|
33 |
git clone https://github.com/Fantasy-AMAP/fantasy-portrait.git
|
34 |
cd fantasy-portrait
|
35 |
```
|
36 |
|
37 |
-
|
38 |
```
|
39 |
apt-get install ffmpeg
|
40 |
-
#
|
41 |
pip install -r requirements.txt
|
42 |
-
#
|
43 |
pip install flash_attn
|
44 |
```
|
45 |
|
46 |
-
### 📦Multi-Expr
|
47 |
-
|
48 |
|
49 |
-
|
50 |
-
|
|
|
51 |
| --------------|-------------------------------------------------------------------------------|-------------------------------|
|
52 |
-
| Wan2.1-I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) |
|
53 |
-
| FantasyPortrait | 🤗 [Huggingface](https://huggingface.co/acvlab/FantasyPortrait/) 🤖 [ModelScope](https://www.modelscope.cn/models/amap_cvlab/FantasyPortrait/) |
|
54 |
|
55 |
-
|
56 |
``` sh
|
57 |
pip install "huggingface_hub[cli]"
|
58 |
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
|
59 |
huggingface-cli download acvlab/FantasyPortrait --local-dir ./models
|
60 |
```
|
61 |
|
62 |
-
|
63 |
``` sh
|
64 |
pip install modelscope
|
65 |
modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
|
66 |
modelscope download amap_cvlab/FantasyPortrait --local_dir ./models
|
67 |
```
|
68 |
|
69 |
-
### 🔑
|
70 |
``` sh
|
71 |
bash infer_single.sh
|
72 |
```
|
73 |
|
74 |
-
### 🔑
|
75 |
-
If you use input image and drive videos with multiple people, you can run as follows:
|
76 |
``` sh
|
77 |
bash infer_multi.sh
|
78 |
```
|
79 |
-
If you use input image with multiple people and different multiple single-human driven videos, you can run as follows:
|
80 |
-
```sh
|
81 |
-
bash infer_multi_diff.sh
|
82 |
-
```
|
83 |
|
84 |
-
### 📦Speed and VRAM Usage
|
85 |
-
We present a detailed table here. The model is tested on a single A100.
|
86 |
|
87 |
-
|
|
|
|
|
|
|
88 |
|-|-|-|-|
|
89 |
-
|torch.bfloat16|None (
|
90 |
-
|torch.bfloat16|7*10**9 (7B)|32.
|
91 |
-
|torch.bfloat16|0|42.
|
92 |
|
93 |
|
94 |
|
95 |
-
## 🧩
|
96 |
-
|
97 |
-
|
98 |
|
99 |
-
## 🔗
|
100 |
-
|
101 |
```
|
102 |
@article{wang2025fantasyportrait,
|
103 |
title={FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers},
|
@@ -107,5 +104,5 @@ If you find this repository useful, please consider giving a star ⭐ and citati
|
|
107 |
}
|
108 |
```
|
109 |
|
110 |
-
##
|
111 |
-
|
|
|
1 |
+
---
|
2 |
+
frameworks:
|
3 |
+
- Pytorch
|
4 |
+
license: Apache License 2.0
|
5 |
+
tasks:
|
6 |
+
- text-to-video-synthesis
|
7 |
+
|
8 |
+
|
9 |
+
# FantasyPortrait:基于表情增强扩散变换器的多角色肖像动画生成
|
10 |
|
11 |
[](https://fantasy-amap.github.io/fantasy-portrait/)
|
12 |
[](https://arxiv.org/abs/2507.12956)
|
13 |
+
[](https://huggingface.co/datasets/acvlab/FantasyPortrait)
|
14 |
[](https://huggingface.co/papers/2507.12956)
|
15 |
|
16 |
+
## 🔥 最新动态!!
|
17 |
+
* 2025年8月10日:我们已发布推理代码、模型权重和数据集。
|
18 |
|
19 |
+
## 演示
|
20 |
+
更多有趣的结果,请访问我们的[网站](https://fantasy-amap.github.io/fantasy-portrait/)。
|
21 |
|
22 |
|  |  |
|
23 |
| :---: | :---: |
|
24 |
|  |  |
|
25 |
|  |  |
|
26 |
|
27 |
+
## 快速开始
|
28 |
+
### 🛠️ 安装
|
29 |
|
30 |
+
克隆仓库:
|
|
|
|
|
|
|
31 |
|
32 |
```
|
33 |
git clone https://github.com/Fantasy-AMAP/fantasy-portrait.git
|
34 |
cd fantasy-portrait
|
35 |
```
|
36 |
|
37 |
+
安装依赖:
|
38 |
```
|
39 |
apt-get install ffmpeg
|
40 |
+
# 确保 torch >= 2.0.0
|
41 |
pip install -r requirements.txt
|
42 |
+
# 注意:必须安装 flash attention
|
43 |
pip install flash_attn
|
44 |
```
|
45 |
|
46 |
+
### 📦 Multi-Expr 数据集
|
47 |
+
我们公开了首个多人肖像面部表情视频数据集 **Multi-Expr Dataset**,请通过以下链接下载:
|
48 |
|
49 |
+
|
50 |
+
### 🧱 模型下载
|
51 |
+
| 模型 | 下载链接 | 说明 |
|
52 |
| --------------|-------------------------------------------------------------------------------|-------------------------------|
|
53 |
+
| Wan2.1-I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | 基础模型
|
54 |
+
| FantasyPortrait | 🤗 [Huggingface](https://huggingface.co/acvlab/FantasyPortrait/) 🤖 [ModelScope](https://www.modelscope.cn/models/amap_cvlab/FantasyPortrait/) | 我们的表情条件权重
|
55 |
|
56 |
+
使用 huggingface-cli 下载模型:
|
57 |
``` sh
|
58 |
pip install "huggingface_hub[cli]"
|
59 |
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
|
60 |
huggingface-cli download acvlab/FantasyPortrait --local-dir ./models
|
61 |
```
|
62 |
|
63 |
+
使用 modelscope-cli 下载模型:
|
64 |
``` sh
|
65 |
pip install modelscope
|
66 |
modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
|
67 |
modelscope download amap_cvlab/FantasyPortrait --local_dir ./models
|
68 |
```
|
69 |
|
70 |
+
### 🔑 单人肖像推理
|
71 |
``` sh
|
72 |
bash infer_single.sh
|
73 |
```
|
74 |
|
75 |
+
### 🔑 多人肖像推理
|
|
|
76 |
``` sh
|
77 |
bash infer_multi.sh
|
78 |
```
|
|
|
|
|
|
|
|
|
79 |
|
|
|
|
|
80 |
|
81 |
+
### 📦 速度与显存占用
|
82 |
+
我们在��提供详细表格。模型在单张A100上进行测试。
|
83 |
+
|
84 |
+
|`torch_dtype`|`num_persistent_param_in_dit`|速度|所需显存|
|
85 |
|-|-|-|-|
|
86 |
+
|torch.bfloat16|None (无限制)|15.5秒/迭代|40G|
|
87 |
+
|torch.bfloat16|7*10**9 (7B)|32.8秒/迭代|20G|
|
88 |
+
|torch.bfloat16|0|42.6秒/迭代|5G|
|
89 |
|
90 |
|
91 |
|
92 |
+
## 🧩 社区贡献
|
93 |
+
我们 ❤️ 来自开源社区的贡献!如果您的工作改进了 FantasyPortrait,请告知我们。
|
94 |
+
您也可以直接发送邮件至 [[email protected]](mailto://[email protected])。我们很乐意引用您的项目,方便大家使用。
|
95 |
|
96 |
+
## 🔗 引用
|
97 |
+
如果本仓库对您有帮助,请考虑给我们一个 star ⭐ 并引用以下论文:
|
98 |
```
|
99 |
@article{wang2025fantasyportrait,
|
100 |
title={FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers},
|
|
|
104 |
}
|
105 |
```
|
106 |
|
107 |
+
## 致谢
|
108 |
+
感谢 [Wan2.1](https://github.com/Wan-Video/Wan2.1)、[PD-FGC](https://github.com/Dorniwang/PD-FGC-inference) 和 [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 开源他们的模型和代码,为本项目提供了宝贵的参考和支持。我们非常感谢他们对开源社区的贡献。
|
configuration.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"framework":"Pytorch","task":"text-to-video-synthesis"}
|
face_det.onnx
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:7ea8de1da304c1459a11f637798bb1140805365aeb3cf6637ca6d61909720aec
|
3 |
+
size 370619
|
face_landmark.onnx
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:463aaa48dd24b9d3b966c76accf9a27f7c590a73a3827d5002a975e46bd54995
|
3 |
+
size 2865751
|
fantasyportrait_model.ckpt
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:95e1b17849ba13766ea21bd9a9a160b115e80f5dc9adf810da600a9b606c81df
|
3 |
+
size 4013364090
|
pd_fpg.pth
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:b34f69d8e6070d1f1e9986a31bc954a8998d45ae74436aa7de128825b3606007
|
3 |
+
size 39542043
|
upload.py
ADDED
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
from huggingface_hub import HfApi
|
2 |
+
|
3 |
+
api = HfApi()
|
4 |
+
local_dir = "/Users/zuojing/wangqiang/FantasyPortrait"
|
5 |
+
repo_id = "acvlab/FantasyPortrait"
|
6 |
+
|
7 |
+
# 上传文件夹
|
8 |
+
api.upload_folder(
|
9 |
+
folder_path=local_dir,
|
10 |
+
repo_id=repo_id,
|
11 |
+
repo_type="model"
|
12 |
+
)
|