Improve model card: Add comprehensive content, fix links, and update metadata
Browse filesThis PR significantly improves the model card for JAM by:
* **Updating Metadata**: Adding the `license: other` tag to accurately reflect the non-commercial use restrictions detailed in the license text, and enriching `tags` with more specific keywords like `song-generation`, `lyrics-to-song`, `flow-matching`, and `direct-preference-optimization` for better discoverability.
* **Enhancing Content**: Replacing the existing content with the comprehensive and well-structured information from the project's GitHub README. This includes:
* Adding the "News" section.
* Adding a dedicated "Paper Abstract" section for quick reference.
* Adding the "The Pipeline" visual explanation.
* **Fixing Links**:
* Correcting the incomplete arXiv badge link to the full arXiv URL.
* Adding a new badge that links directly to the Hugging Face paper page (`https://huggingface.co/papers/2507.20880`).
* Updating relative image paths (e.g., `jam-teaser.png`, `jam.png`) to absolute GitHub raw URLs to ensure images render correctly.
* Converting relative license document links (`LICENSE.md`, `STABILITY_AI_COMMUNITY_LICENSE.md`) to absolute GitHub raw URLs for reliable access.
* **Completing Citation**: Updating the BibTeX citation with `eprint` and `url` fields for a more complete reference.
These changes provide a much richer, more accurate, and more robust model card, making it easier for users to understand, use, and cite the model.
@@ -1,6 +1,7 @@
|
|
1 |
---
|
2 |
language:
|
3 |
- en
|
|
|
4 |
metrics:
|
5 |
- PER
|
6 |
- WER
|
@@ -9,32 +10,50 @@ metrics:
|
|
9 |
- MuQ
|
10 |
- FAD
|
11 |
pipeline_tag: text-to-audio
|
12 |
-
library_name: diffusers
|
13 |
tags:
|
14 |
- music
|
15 |
- art
|
|
|
|
|
|
|
|
|
|
|
16 |
---
|
|
|
17 |
<div align="center">
|
18 |
<img src="https://declare-lab.github.io/jamify-logo-new.png" width="200"/ >
|
19 |
<br/>
|
20 |
<h1>JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment</h1>
|
21 |
<br/>
|
22 |
|
23 |
-
[](https://arxiv.org/abs/2507.) [](https://huggingface.co/declare-lab/JAM-0.5) [](https://declare-lab.github.io/jamify) [](https://github.com/declare-lab/jamify)
|
25 |
|
26 |
</div>
|
27 |
|
28 |
JAM is a rectified flow-based model for lyrics-to-song generation that addresses the lack of fine-grained word-level controllability in existing lyrics-to-song models. Built on a compact 530M-parameter architecture with 16 LLaMA-style Transformer layers as the Diffusion Transformer (DiT) backbone, JAM enables precise vocal control that musicians desire in their workflows. Unlike previous models, JAM provides word and phoneme-level timing control, allowing musicians to specify the exact placement of each vocal sound for improved rhythmic flexibility and expressive timing.
|
29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
## Features
|
31 |
|
|
|
|
|
32 |
- **Fine-grained Word and Phoneme-level Timing Control**: The first model to provide word-level timing and duration control in song generation, enabling precise prosody control for musicians
|
33 |
- **Compact 530M Parameter Architecture**: Less than half the size of existing models, enabling faster inference with reduced resource requirements
|
34 |
- **Enhanced Lyric Fidelity**: Achieves over 3× reduction in Word Error Rate (WER) and Phoneme Error Rate (PER) compared to prior work through precise phoneme boundary attention
|
35 |
- **Global Duration Control**: Controllable duration up to 3 minutes and 50 seconds.
|
36 |
- **Aesthetic Alignment through Direct Preference Optimization**: Iterative refinement using synthetic preference datasets to better align with human aesthetic preferences, eliminating manual annotation requirements
|
37 |
|
|
|
|
|
|
|
|
|
38 |
## JAM Samples
|
39 |
|
40 |
Check out the example generated music in the `generated_examples/` folder to hear what JAM can produce:
|
@@ -262,7 +281,10 @@ If you use JAM in your research, please cite:
|
|
262 |
@misc{jam2024,
|
263 |
title={JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment},
|
264 |
author={Renhang Liu and Chia-Yu Hung and Navonil Majumder and Taylor Gautreaux and Amir Ali Bagherzadeh and Chuan Li and Dorien Herremans and Soujanya Poria},
|
265 |
-
year={2025}
|
|
|
|
|
|
|
266 |
}
|
267 |
```
|
268 |
|
@@ -283,7 +305,7 @@ If you use JAM in your research, please cite:
|
|
283 |
### Responsibility
|
284 |
Responsibility for the use of the model and its outputs lies entirely with the end user, who must ensure all uses comply with applicable legal and ethical standards.
|
285 |
|
286 |
-
For complete license terms, see [LICENSE.md](LICENSE.md) and [STABILITY_AI_COMMUNITY_LICENSE.md](STABILITY_AI_COMMUNITY_LICENSE.md).
|
287 |
|
288 |
For questions, concerns, or collaboration inquiries, please contact the Project Jamify team via the official repository.
|
289 |
|
|
|
1 |
---
|
2 |
language:
|
3 |
- en
|
4 |
+
library_name: diffusers
|
5 |
metrics:
|
6 |
- PER
|
7 |
- WER
|
|
|
10 |
- MuQ
|
11 |
- FAD
|
12 |
pipeline_tag: text-to-audio
|
|
|
13 |
tags:
|
14 |
- music
|
15 |
- art
|
16 |
+
- song-generation
|
17 |
+
- lyrics-to-song
|
18 |
+
- flow-matching
|
19 |
+
- direct-preference-optimization
|
20 |
+
license: other
|
21 |
---
|
22 |
+
|
23 |
<div align="center">
|
24 |
<img src="https://declare-lab.github.io/jamify-logo-new.png" width="200"/ >
|
25 |
<br/>
|
26 |
<h1>JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment</h1>
|
27 |
<br/>
|
28 |
|
29 |
+
[](https://arxiv.org/abs/2507.20880) [](https://huggingface.co/papers/2507.20880) [](https://huggingface.co/declare-lab/JAM-0.5) [](https://declare-lab.github.io/jamify) [](https://github.com/declare-lab/jamify)
|
31 |
|
32 |
</div>
|
33 |
|
34 |
JAM is a rectified flow-based model for lyrics-to-song generation that addresses the lack of fine-grained word-level controllability in existing lyrics-to-song models. Built on a compact 530M-parameter architecture with 16 LLaMA-style Transformer layers as the Diffusion Transformer (DiT) backbone, JAM enables precise vocal control that musicians desire in their workflows. Unlike previous models, JAM provides word and phoneme-level timing control, allowing musicians to specify the exact placement of each vocal sound for improved rhythmic flexibility and expressive timing.
|
35 |
|
36 |
+
## News
|
37 |
+
> 📣 29/07/25: We have released JAM-0.5, the first version of the AI song generator from Project Jamify!
|
38 |
+
|
39 |
+
## Paper Abstract
|
40 |
+
|
41 |
+
Diffusion and flow-matching models have revolutionized automatic text-to-audio generation in recent times. These models are increasingly capable of generating high quality and faithful audio outputs capturing to speech and acoustic events. However, there is still much room for improvement in creative audio generation that primarily involves music and songs. Recent open lyrics-to-song models, such as, DiffRhythm, ACE-Step, and LeVo, have set an acceptable standard in automatic song generation for recreational use. However, these models lack fine-grained word-level controllability often desired by musicians in their workflows. To the best of our knowledge, our flow-matching-based JAM is the first effort toward endowing word-level timing and duration control in song generation, allowing fine-grained vocal control. To enhance the quality of generated songs to better align with human preferences, we implement aesthetic alignment through Direct Preference Optimization, which iteratively refines the model using a synthetic dataset, eliminating the need or manual data annotations. Furthermore, we aim to standardize the evaluation of such lyrics-to-song models through our public evaluation dataset JAME. We show that JAM outperforms the existing models in terms of the music-specific attributes.
|
42 |
+
|
43 |
## Features
|
44 |
|
45 |
+

|
46 |
+
|
47 |
- **Fine-grained Word and Phoneme-level Timing Control**: The first model to provide word-level timing and duration control in song generation, enabling precise prosody control for musicians
|
48 |
- **Compact 530M Parameter Architecture**: Less than half the size of existing models, enabling faster inference with reduced resource requirements
|
49 |
- **Enhanced Lyric Fidelity**: Achieves over 3× reduction in Word Error Rate (WER) and Phoneme Error Rate (PER) compared to prior work through precise phoneme boundary attention
|
50 |
- **Global Duration Control**: Controllable duration up to 3 minutes and 50 seconds.
|
51 |
- **Aesthetic Alignment through Direct Preference Optimization**: Iterative refinement using synthetic preference datasets to better align with human aesthetic preferences, eliminating manual annotation requirements
|
52 |
|
53 |
+
## The Pipeline
|
54 |
+
|
55 |
+

|
56 |
+
|
57 |
## JAM Samples
|
58 |
|
59 |
Check out the example generated music in the `generated_examples/` folder to hear what JAM can produce:
|
|
|
281 |
@misc{jam2024,
|
282 |
title={JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment},
|
283 |
author={Renhang Liu and Chia-Yu Hung and Navonil Majumder and Taylor Gautreaux and Amir Ali Bagherzadeh and Chuan Li and Dorien Herremans and Soujanya Poria},
|
284 |
+
year={2025},
|
285 |
+
eprint={2507.20880},
|
286 |
+
archivePrefix={arXiv},
|
287 |
+
url={https://huggingface.co/papers/2507.20880},
|
288 |
}
|
289 |
```
|
290 |
|
|
|
305 |
### Responsibility
|
306 |
Responsibility for the use of the model and its outputs lies entirely with the end user, who must ensure all uses comply with applicable legal and ethical standards.
|
307 |
|
308 |
+
For complete license terms, see [LICENSE.md](https://github.com/declare-lab/jamify/blob/main/LICENSE.md) and [STABILITY_AI_COMMUNITY_LICENSE.md](https://github.com/declare-lab/jamify/blob/main/STABILITY_AI_COMMUNITY_LICENSE.md).
|
309 |
|
310 |
For questions, concerns, or collaboration inquiries, please contact the Project Jamify team via the official repository.
|
311 |
|