README.md

---
language:
- ja
base_model:
- webbigdata/VoiceCore
tags:
- tts
- vllm
---

# VoiceCore_smoothquant

[webbigdata/VoiceCore](https://huggingface.co/webbigdata/VoiceCore)をvLLMで高速に動かすためにgptq(W4A16)量子化したモデルです  
詳細は[webbigdata/VoiceCore](https://huggingface.co/webbigdata/VoiceCore)のモデルカードを御覧ください  

This is a model quantized using gptq(W4A16) to run [webbigdata/VoiceCore](https://huggingface.co/webbigdata/VoiceCore) at high speed using vLLM.  
See the [webbigdata/VoiceCore](https://huggingface.co/webbigdata/VoiceCore) model card for details.  


## Install/Setup

[vLLMはAMDのGPUでも動作する](https://docs.vllm.ai/en/v0.6.5/getting_started/amd-installation.html)そうですがチェックは出来ていません。  
Mac(CPU)でも動くようですが、[gguf版](https://huggingface.co/webbigdata/VoiceCore_gguf)を使った方が早いかもしれません  

vLLM seems to work with [AMD GPUs](https://docs.vllm.ai/en/v0.6.5/getting_started/amd-installation.html), but I haven't checked.  
It also seems to work with Mac (CPU), but [gguf version](https://huggingface.co/webbigdata/VoiceCore_gguf) seems to be better.  

以下はLinuxのNvidia GPU版のセットアップ手順です  
Below are the setup instructions for the Nvidia GPU version of Linux.  

```
python3 -m venv VL
source VL/bin/activate
pip install vllm
pip install snac
pip install numpy==1.26.4
pip install transformers==4.53.2
```

## Sample script
```
import torch
import scipy.io.wavfile as wavfile
from transformers import AutoTokenizer
from snac import SNAC
from vllm import LLM, SamplingParams

QUANTIZED_MODEL_PATH = "webbigdata/VoiceCore_gptq"
prompts = [
     "テストです",
     "ジーピーティーキュー、問題なく動いてますかね？あ～、笑い声が上手く表現できなくなっちゃってますかね、仕方ないか、えへへ"
]
chosen_voice = "matsukaze_male[neutral]"

print("Loading tokenizer and preparing inputs...")
tokenizer = AutoTokenizer.from_pretrained(QUANTIZED_MODEL_PATH)
prompts_ = [(f"{chosen_voice}: " + p) if chosen_voice else p for p in prompts]
start_token, end_tokens = [128259], [128009, 128260, 128261]
all_prompt_token_ids = []
for prompt in prompts_:
  input_ids = tokenizer.encode(prompt)
  final_token_ids = start_token + input_ids + end_tokens
  all_prompt_token_ids.append(final_token_ids)
print("Inputs prepared successfully.")

print(f"Loading SmoothQuant model with vLLM from: {QUANTIZED_MODEL_PATH}")
llm = LLM(
    model=QUANTIZED_MODEL_PATH,
    trust_remote_code=True,
    max_model_len=10000,    # メモリ不足になる場合は減らしてください f you run out of memory, reduce it.  
    #gpu_memory_utilization=0.9 # 「最大GPUメモリの何割を使うか？」適宜調整してください  "What percentage of the maximum GPU memory should be used?" Adjust accordingly.
)
sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.90,
    repetition_penalty=1.1,
    max_tokens=8192, # max_tokens + input_prompt <= max_model_len
    stop_token_ids=[128258]
)
print("vLLM model loaded.")

print("Generating audio tokens with vLLM...")
outputs = llm.generate(prompt_token_ids=all_prompt_token_ids, sampling_params=sampling_params)
print("Generation complete.")

# GPUの方が早いがvllmが大きくメモリ確保していると失敗するため  GPU is faster, but if vllm allocates a lot of memory it will fail to run.
print("Loading SNAC decoder to CPU...")
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
snac_model.to("cpu") 
print("SNAC model loaded.")

print("Decoding tokens to audio...")
audio_start_token = 128257

def redistribute_codes(code_list):
  layer_1, layer_2, layer_3 = [], [], []
  for i in range(len(code_list) // 7):
    layer_1.append(code_list[7*i])
    layer_2.append(code_list[7*i+1] - 4096)
    layer_3.append(code_list[7*i+2] - (2*4096))
    layer_3.append(code_list[7*i+3] - (3*4096))
    layer_2.append(code_list[7*i+4] - (4*4096))
    layer_3.append(code_list[7*i+5] - (5*4096))
    layer_3.append(code_list[7*i+6] - (6*4096))

  codes = [torch.tensor(layer).unsqueeze(0)
           for layer in [layer_1, layer_2, layer_3]]

  audio_hat = snac_model.decode(codes)
  return audio_hat

code_lists = []
for output in outputs:
    generated_token_ids = output.outputs[0].token_ids
    generated_tensor = torch.tensor([generated_token_ids])
    token_indices = (generated_tensor == audio_start_token).nonzero(as_tuple=True)
    if len(token_indices[1]) > 0:
        cropped_tensor = generated_tensor[:, token_indices[1][-1].item() + 1:]
    else:
        cropped_tensor = generated_tensor

    masked_row = cropped_tensor.squeeze()
    row_length = masked_row.size(0)
    new_length = (row_length // 7) * 7
    trimmed_row = masked_row[:new_length]
    code_list = [t.item() - 128266 for t in trimmed_row]
    code_lists.append(code_list)

for i, code_list in enumerate(code_lists):
    if i >= len(prompts): break

    print(f"Processing audio for prompt: '{prompts[i]}'")
    samples = redistribute_codes(code_list)
    sample_np = samples.detach().squeeze().numpy()

    safe_prompt = "".join(c for c in prompts[i] if c.isalnum() or c in (' ', '_')).rstrip()
    filename = f"audio_final_{i}_{safe_prompt[:20].replace(' ', '_')}.wav"

    wavfile.write(filename, 24000, sample_np)
    print(f"Saved audio to: {filename}")
```


## Streaming sample

vLLMをサーバーとして動作させてストリーミングでアクセスさせ、クライアントが逐次再生するデモです。  
品質は劣化してしまいますがRTX 4060くらいの性能をもつGPUなら疑似リアルタイム再生が実現できます。  
理想は雑音が生成されないタイミングで生成する事ですが、まだ実現出来ておらず、実証実験レベルとお考え下さい 

### Sever side command 
(Linux server前提)
```
python3 -m vllm.entrypoints.openai.api_server --model VoiceCore_gptq --host 0.0.0.0 --port 8000 --max-model-len 9000
```
### Client side scripyt
(Windows前提)  
SERVER_URLを書き換えてください  
```
import torch
from transformers import AutoTokenizer
from snac import SNAC
import requests
import json
import sounddevice as sd
import numpy as np
import queue
import threading

# --- サーバー設定とモデルの準備 (変更なし) ---
SERVER_URL = "http://192.168.1.16:8000/v1/completions"
TOKENIZER_PATH = "webbigdata/VoiceCore_gptq"
MODEL_NAME = "VoiceCore_gptq"

prompts = [
     "テストです",
     "ジーピーティーキュー、問題なく動いてますかね？圧縮しすぎると別人の声になっちゃう事があるんですよね、ふふふ"
]
chosen_voice = "matsukaze_male[neutral]"

print("Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH)
start_token, end_tokens = [128259], [128009, 128260, 128261]

print("Loading SNAC decoder to CPU...")
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
snac_model.to("cpu")
print("SNAC model loaded.")
audio_start_token = 128257

def redistribute_codes(code_list):
    if len(code_list) % 7 != 0: return torch.tensor([])
    layer_1, layer_2, layer_3 = [], [], []
    for i in range(len(code_list) // 7):
        layer_1.append(code_list[7*i])
        layer_2.append(code_list[7*i+1] - 4096)
        layer_3.append(code_list[7*i+2] - (2*4096)); layer_3.append(code_list[7*i+3] - (3*4096))
        layer_2.append(code_list[7*i+4] - (4*4096)); layer_3.append(code_list[7*i+5] - (5*4096))
        layer_3.append(code_list[7*i+6] - (6*4096))
    codes = [torch.tensor(layer).unsqueeze(0) for layer in [layer_1, layer_2, layer_3]]
    return snac_model.decode(codes)


def audio_playback_worker(q, stream):
    while True:
        data = q.get()
        if data is None:
            break
        stream.write(data)

for i, prompt in enumerate(prompts):
    print("\n" + "="*50)
    print(f"Processing prompt ({i+1}/{len(prompts)}): '{prompt}'")
    print("="*50)

    prompt_ = (f"{chosen_voice}: " + prompt) if chosen_voice else prompt
    input_ids = tokenizer.encode(prompt_)
    final_token_ids = start_token + input_ids + end_tokens
    
    payload = {
        "model": MODEL_NAME, "prompt": final_token_ids,
        "max_tokens": 8192, "temperature": 0.6, "top_p": 0.90,
        "repetition_penalty": 1.1, "stop_token_ids": [128258],
        "stream": True
    }

    token_buffer = []
    found_audio_start = False
    CHUNK_SIZE = 28 
    
    audio_queue = queue.Queue()
    playback_stream = sd.OutputStream(samplerate=24000, channels=1, dtype='float32')
    playback_stream.start()
    
    playback_thread = threading.Thread(target=audio_playback_worker, args=(audio_queue, playback_stream))
    playback_thread.start()

    try:
        response = requests.post(SERVER_URL, headers={"Content-Type": "application/json"}, json=payload, stream=True)
        response.raise_for_status()

        for line in response.iter_lines():
            if line:
                decoded_line = line.decode('utf-8')
                if decoded_line.startswith('data: '):
                    content = decoded_line[6:]
                    if content == '[DONE]':
                        break
                    
                    try:
                        chunk = json.loads(content)
                        text_chunk = chunk['choices'][0]['text']
                        if text_chunk:
                            token_buffer.extend(tokenizer.encode(text_chunk, add_special_tokens=False))
                        
                        if not found_audio_start:
                            try:
                                start_index = token_buffer.index(audio_start_token)
                                token_buffer = token_buffer[start_index + 1:]
                                found_audio_start = True
                                print("Audio start token found. Starting playback...")
                            except ValueError:
                                continue
                        
                        while len(token_buffer) >= CHUNK_SIZE:
                            tokens_to_process = token_buffer[:CHUNK_SIZE]
                            token_buffer = token_buffer[CHUNK_SIZE:]
                            
                            code_list = [t - 128266 for t in tokens_to_process]
                            samples = redistribute_codes(code_list)
                            
                            if samples.numel() > 0:
                                sample_np = samples.detach().squeeze().numpy()
                                audio_queue.put(sample_np)

                    except (json.JSONDecodeError, Exception) as e:
                        print(f"処理中にエラー: {e}")

        if found_audio_start and token_buffer:
            remaining_length = (len(token_buffer) // 7) * 7
            if remaining_length > 0:
                tokens_to_process = token_buffer[:remaining_length]
                code_list = [t - 128266 for t in tokens_to_process]
                samples = redistribute_codes(code_list)
                if samples.numel() > 0:
                    sample_np = samples.detach().squeeze().numpy()
                    audio_queue.put(sample_np)

    except requests.exceptions.RequestException as e:
        print(f"サーバーへのリクエストでエラーが発生しました: {e}")
    finally:
        audio_queue.put(None)
        playback_thread.join()
        playback_stream.stop()
        playback_stream.close()
        print("Playback finished for this prompt.")

print("\nAll processing complete!")
```