Update README.md

README.md

@@ -1,199 +1,81 @@
 ---
 library_name: transformers
-tags: []
+license: llama3.2
+datasets:
+- slprl/sTinyStories
+language:
+- en
+base_model:
+- meta-llama/Llama-3.2-3B
+pipeline_tag: audio-to-audio
 ---
 
-# Model Card for Model ID
+# Scaling Analysis of Interleaved Speech-Text Language Models
+
+The model was presented in the paper [Scaling Analysis of Interleaved Speech-Text Language Models](https://arxiv.org/abs/).
 
-<!-- Provide a quick summary of what the model is/does. -->
+# Paper abstract
+Existing Speech Language Model (SLM) scaling analysis paints a bleak picture. They predict that SLMs require much more compute and data 
+compared to text, leading some to question the feasibility of training high-quality SLMs. However, modern SLMs are often initialised from 
+pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. This raises the question - _Do interleaved SLMs scale more efficiently than textless-SLMs?_ 
+In this paper we answer a resounding _yes!_ We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the 
+scaling trends. We see that under this setup SLMs scale more efficiently with compute. Additionally, our results indicate that the 
+scaling-dynamics are significantly different than textless-SLMs, suggesting one should allocate notably more of the compute budget for 
+increasing model size over training tokens. We also study the role of synthetic data and TextLM model families in unlocking this potential. 
+Results suggest, that our scaled up model achieves comparable performance with leading models on speech semantic metrics while using less 
+compute and data than other approaches.
 
+# Model Card for Model ID
+This is a Speech Language Model (SLM) trained for generating speech or text continuations over discrete [Hubert tokens](https://huggingface.co/slprl/mhubert-base-25hz) given speech-text prompts.
 
 
 ## Model Details
 
 ### Model Description
+This Speech Language Model, introduced in ["Scaling Analysis of Interleaved Speech-Text Language Models"](https://arxiv.org/abs/), focuses on scaling analysis of interleaved speech-text SLMs. 
+It was fine-tuned from [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B) by extending its vocabulary with 500 speech tokens extracted from 
+the 11-th layer of [mhubert-25hz](https://huggingface.co/slprl/mhubert-base-25hz).
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** [SLP-RL](https://huggingface.co/slprl)
+- **Model type:** SpeechLM
+- **License:** Llama3.2 licensed
+- **Finetuned from model:** [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B)
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** [https://github.com/slp-rl/slamkit](https://github.com/slp-rl/slamkit)
+- **Paper:** [https://arxiv.org/abs/](https://arxiv.org/abs/)
+- **Demo:** [https://pages.cs.huji.ac.il/adiyoss-lab/sims/](https://pages.cs.huji.ac.il/adiyoss-lab/sims/)
 
 ## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+This base SpeechLM can be used to generate continuations for speech segments, or cross-modal e.g generate a text contiuation to a speech prompt, or as a base for further tuning. See the _SlamKit_
+[codebase](https://github.com/slp-rl/slamkit) for more details on usage, and checkout the [demo page](https://pages.cs.huji.ac.il/adiyoss-lab/sims/) for some generation examples
 
 ### Out-of-Scope Use
+This model was trained on diverse speech datasets, as such the outputs should not be treated as factual in any way.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
 ## How to Get Started with the Model
+We refer users to the official repository for full usage explanations - [github](https://github.com/slp-rl/slamkit).
 
-Use the code below to get started with the model.
-
-[More Information Needed]
 
 ## Training Details
+We highly encourage users to read the full [paper](https://arxiv.org/abs/2502.15814), for full training details.
 
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
 
 ### Compute Infrastructure
-
-[More Information Needed]
-
 #### Hardware
-
-[More Information Needed]
+This model was trained using 8 Nvidia A100 GPUs.
 
 #### Software
+The model was trained using the [*SlamKit*](https://github.com/slp-rl/slamkit) codebase which builds upon 🤗transformers extending it to support
+easy and efficient training of Speech Language Models.
 
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Citation
 
 **BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+```
+@misc{maimon2025scaling,
+      soon 
+}
+```