diff --git a/demos/audio/README.md b/demos/audio/README.md index 1ae8ef7242..ae1ca5b064 100644 --- a/demos/audio/README.md +++ b/demos/audio/README.md @@ -30,15 +30,15 @@ See the [T2s calculator documentation](../../docs/speech_generation/reference.md **Deploying with Docker** ```bash -mkdir models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device CPU --task text2speech +mkdir -p ${HOME}/models +docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device CPU ``` **Deploying on Bare Metal** ```bat mkdir c:\models -ovms --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path c:\models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device CPU --task text2speech +ovms --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path c:\models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device CPU ``` ### Request Generation @@ -107,33 +107,21 @@ Many variances of Whisper models can be deployed in a single command by using pr In this demo we will use OpenVINO/whisper-large-v3-turbo-fp16-ov, which is a fine-tuned version of Whisper large-v3. :::{dropdown} **Deploying with Docker** - -Select deployment option depending on how you prepared models in the previous step. - -**CPU** - -Running this command starts the container with CPU only target device: -```bash -mkdir -p models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --task speech2text --source_model OpenVINO/whisper-large-v3-turbo-fp16-ov --model_name whisper-large-v3-turbo-fp16-ov --model_repository_path /models --target_device CPU -``` -**GPU** - -In case you want to use GPU device to run the generation, add extra docker parameters `--device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1)` -to `docker run` command, use the image with GPU support. -It can be applied using the commands below: ```bash -mkdir -p models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -v $(pwd)/models:/models:rw openvino/model_server:latest-gpu --rest_port 8000 --task speech2text --source_model OpenVINO/whisper-large-v3-turbo-fp16-ov --model_name whisper-large-v3-turbo-fp16-ov --model_repository_path /models --target_device GPU +mkdir -p ${HOME}/models +# in case GPU is available +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run -d ${GPU_ARGS} -u $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly --rest_port 8000 --source_model OpenVINO/whisper-large-v3-turbo-fp16-ov --model_name whisper-large-v3-turbo-fp16-ov --model_repository_path /models ``` ::: :::{dropdown} **Deploying on Bare Metal** -If you run on GPU make sure to have appropriate drivers installed, so the device is accessible for the model server. +The same command can be used for CPU and GPU deployments. OVMS will auto-detect device settings when `--target_device` is not provided. ```bat -ovms --rest_port 8000 --task speech2text --source_model OpenVINO/whisper-large-v3-turbo-fp16-ov --model_name whisper-large-v3-turbo-fp16-ov --model_repository_path models --target_device GPU +mkdir c:\models +ovms --rest_port 8000 --source_model OpenVINO/whisper-large-v3-turbo-fp16-ov --model_name whisper-large-v3-turbo-fp16-ov --model_repository_path c:\models ``` ::: @@ -254,48 +242,29 @@ Prepare export script and dependencies: ```console curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py -o export_model.py pip install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt -mkdir -p models +mkdir -p ${HOME}/models ``` Export Speech-to-Text model with word timestamps enabled. - -**CPU** - ```console -python export_model.py speech2text --source_model openai/whisper-large-v3-turbo --weight-format fp16 --model_name whisper-large-v3-turbo-word-ts --config_file_path models/config.json --model_repository_path models --overwrite_models --enable_word_timestamps --target_device CPU -``` - -**GPU** - -```console -python export_model.py speech2text --source_model openai/whisper-large-v3-turbo --weight-format fp16 --model_name whisper-large-v3-turbo-word-ts --config_file_path models/config.json --model_repository_path models --overwrite_models --enable_word_timestamps --target_device GPU +python export_model.py speech2text --source_model openai/whisper-large-v3-turbo --weight-format fp16 --model_name whisper-large-v3-turbo-word-ts --config_file_path ${HOME}/models/config.json --model_repository_path ${HOME}/models --overwrite_models --enable_word_timestamps ``` :::{dropdown} **Deploying with Docker** - -Select deployment option depending on how you prepared models in the previous step. - -**CPU** - -```bash -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --config_path /models/config.json -``` - -**GPU** - -In case you want to use GPU device to run the generation, add extra docker parameters `--device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1)` -to `docker run` command, use the image with GPU support. -It can be applied using the commands below: - ```bash -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -v $(pwd)/models:/models:rw openvino/model_server:latest-gpu --rest_port 8000 --config_path /models/config.json +# in case GPU is available +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run -d ${GPU_ARGS} -u $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly --rest_port 8000 --config_path /models/config.json ``` ::: :::{dropdown} **Deploying on Bare Metal** +The same command can be used for CPU and GPU deployments. + ```bat -ovms --rest_port 8000 --config_path models/config.json +mkdir c:\models +ovms --rest_port 8000 --config_path c:\models\config.json ``` ::: @@ -402,34 +371,20 @@ To prepare an audio file with speech in a language other than English, e.g. Span **Deploying with Docker** -**CPU** - ```bash -mkdir -p models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device CPU --task text2speech -``` - -**GPU** - -```bash -mkdir -p models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -v $(pwd)/models:/models:rw openvino/model_server:latest-gpu --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device GPU --task text2speech +mkdir -p ${HOME}/models +# in case GPU is available +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run -d ${GPU_ARGS} -u $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --model_name Kokoro-82M-OpenVINO-FP16-OVMS ``` **Deploying on Bare Metal** -**CPU** - -```bat -mkdir models -ovms --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device CPU --task text2speech -``` - -**GPU** +The same command can be used for CPU and GPU deployments. OVMS will auto-detect device settings when `--target_device` is not provided. ```bat -mkdir models -ovms --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path models --model_name Kokoro-82M-OpenVINO-FP16-OVMS --target_device GPU --task text2speech +mkdir c:\models +ovms --rest_port 8000 --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path c:\models --model_name Kokoro-82M-OpenVINO-FP16-OVMS ``` For non-English Kokoro input, set the `language` field explicitly. @@ -443,38 +398,21 @@ Whisper models can be deployed in a single command by using pre-configured model Here is an example of OpenVINO/whisper-large-v3-fp16-ov deployment: :::{dropdown} **Deploying with Docker** - -Select deployment option depending on how you prepared models in the previous step. - -**CPU** - -Running this command starts the container with CPU only target device: -```bash -mkdir -p models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 -v $(pwd)/models:/models:rw openvino/model_server:latest --rest_port 8000 --source_model OpenVINO/whisper-large-v3-fp16-ov --model_repository_path /models --model_name whisper-large-v3-fp16-ov --task speech2text --target_device CPU -``` -**GPU** - -In case you want to use GPU device to run the generation, add extra docker parameters `--device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1)` -to `docker run` command, use the image with GPU support. -It can be applied using the commands below: ```bash -mkdir -p models -docker run -d -u $(id -u):$(id -g) --rm -p 8000:8000 --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -v $(pwd)/models:/models:rw openvino/model_server:latest-gpu --rest_port 8000 --source_model OpenVINO/whisper-large-v3-fp16-ov --model_repository_path /models --model_name whisper-large-v3-fp16-ov --task speech2text --target_device GPU +mkdir -p ${HOME}/models +# in case GPU is available +export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi) +docker run -d ${GPU_ARGS} -u $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly --rest_port 8000 --source_model OpenVINO/whisper-large-v3-fp16-ov --model_repository_path /models --model_name whisper-large-v3-fp16-ov ``` ::: :::{dropdown} **Deploying on Bare Metal** -If you run on GPU make sure to have appropriate drivers installed, so the device is accessible for the model server. +The same command can be used for CPU and GPU deployments. OVMS will auto-detect device settings when `--target_device` is not provided. ```bat -mkdir models -ovms --rest_port 8000 --source_model OpenVINO/whisper-large-v3-fp16-ov --model_repository_path models --model_name whisper-large-v3-fp16-ov --task speech2text --target_device CPU -``` -or -```bat -ovms --rest_port 8000 --source_model OpenVINO/whisper-large-v3-fp16-ov --model_repository_path models --model_name whisper-large-v3-fp16-ov --task speech2text --target_device GPU +mkdir c:\models +ovms --rest_port 8000 --source_model OpenVINO/whisper-large-v3-fp16-ov --model_repository_path c:\models --model_name whisper-large-v3-fp16-ov ``` ::: diff --git a/demos/benchmark/v3/requirements.txt b/demos/benchmark/v3/requirements.txt index 3272bc11c4..e51e5301eb 100644 --- a/demos/benchmark/v3/requirements.txt +++ b/demos/benchmark/v3/requirements.txt @@ -1,7 +1,7 @@ datasets==3.6.0 dataclasses==0.6 transformers<5.1 -numpy==2.3.5 +numpy==2.2.6 tqdm==4.67.1 sentencepiece==0.2.1 soundfile==0.13.1 diff --git a/demos/integration_with_OpenWebUI/README.md b/demos/integration_with_OpenWebUI/README.md index f077c98de9..be0409f28a 100644 --- a/demos/integration_with_OpenWebUI/README.md +++ b/demos/integration_with_OpenWebUI/README.md @@ -461,11 +461,10 @@ Then it's ready to use. In new chat it's possible to toggle **Code Interpreter** ### Step 1: Models Preparation -Start by downloading `export_model.py` script and run it to download and quantize the model for speech generation: +Start by pulling the pre-exported Kokoro model with OVMS and adding it to the server config: ```console -curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py -o export_model.py -pip3 install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt -python export_model.py text2speech --source_model microsoft/speecht5_tts --weight-format fp32 --model_name microsoft/speecht5_tts --config_file_path models/config.json --model_repository_path models --vocoder microsoft/speecht5_hifigan +docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) openvino/model_server:weekly --pull --source_model luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_repository_path /models --target_device GPU +docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --add_to_config --config_path /models/config.json --model_path luis-castillo/Kokoro-82M-OpenVINO-FP16-OVMS --model_name Kokoro-82M-OpenVINO-FP16-OVMS ``` Next, download and add to config model for transcription: @@ -496,7 +495,7 @@ docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_serve * URL: `http://localhost:8000/v3` * Set Engine type to `OpenAI` * STT Model: `OpenVINO/whisper-base-fp16-ov` - * TTS Model: `microsoft/speecht5_tts` + * TTS Model: `Kokoro-82M-OpenVINO-FP16-OVMS` * Put anything in API key 3. Click **Save** diff --git a/docs/clients_genai.md b/docs/clients_genai.md index bc6dbb55af..329c439e0a 100644 --- a/docs/clients_genai.md +++ b/docs/clients_genai.md @@ -463,8 +463,8 @@ speech_file_path = Path(__file__).parent / "speech.wav" client = OpenAI(base_url=url, api_key="not_used") with client.audio.speech.with_streaming_response.create( - model="microsoft/speecht5_tts", - voice="unused", + model="Kokoro-82M-OpenVINO-FP16-OVMS", + voice="af_alloy", input=prompt ) as response: response.stream_to_file(speech_file_path) @@ -475,7 +475,7 @@ with client.audio.speech.with_streaming_response.create( ```text curl http://localhost:8000/v3/audio/speech \ -H "Content-Type: application/json" \ - -d "{\"model\": \"microsoft/speecht5_tts\", \"input\": \"The quick brown fox jumped over the lazy dog\"}" \ + -d "{\"model\": \"Kokoro-82M-OpenVINO-FP16-OVMS\", \"voice\": \"af_alloy\", \"input\": \"The quick brown fox jumped over the lazy dog\"}" \ -o speech.wav ``` ::: diff --git a/docs/genai.md b/docs/genai.md index e6fe7f54f5..0f4ed330b3 100644 --- a/docs/genai.md +++ b/docs/genai.md @@ -57,7 +57,7 @@ Check also the demo of [transcription](../demos/audio/README.md#transcription) a ## Speech generation The implementation is compatible with the OpenAI API for [audio/speech](./model_server_rest_api_text_to_speech.md). -It supports microsoft/speecht5_tts model. +It supports Kokoro speech generation models, for example `Kokoro-82M-OpenVINO-FP16-OVMS`. Check also the demo of [speech generation](../demos/audio/README.md#speech-generation). diff --git a/docs/model_server_rest_api_text_to_speech.md b/docs/model_server_rest_api_text_to_speech.md index 98701efb55..a9a12cffb3 100644 --- a/docs/model_server_rest_api_text_to_speech.md +++ b/docs/model_server_rest_api_text_to_speech.md @@ -16,8 +16,9 @@ Request body must be in JSON format, and the request must have `Content-Type: ap curl http://localhost:8000/v3/audio/speech \ -H "Content-Type: application/json" \ -d '{ - "model": "microsoft/speecht5_tts", - "input": "The quick brown fox jumped over the lazy dog.", + "model": "Kokoro-82M-OpenVINO-FP16-OVMS", + "voice": "af_alloy", + "input": "The quick brown fox jumped over the lazy dog." }' \ -o speech.wav ``` diff --git a/docs/parameters.md b/docs/parameters.md index b20c93fc9c..e30ba5cb0c 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -118,7 +118,6 @@ When pulling models outside of OpenVINO organization the optimum-cli api is used |------------------------------|--------------|---------------------------------------------------------------------------------------------------------------| | `--extra_quantization_params`| `string` | Add advanced quantization parameters. Check [optimum-intel](https://github.com/huggingface/optimum-intel) documentation. Example: `--sym --group-size -1 --ratio 1.0 --awq --scale-estimation --dataset wikitext2` | | `--weight-format` | `string` | Model precision used in optimum-cli export with conversion. Default `int8`. | -| `--vocoder` | `string` | The vocoder model to use for text2speech. For example `microsoft/speecht5_hifigan`. | There are also additional environment variables that may change the behavior of pulling: @@ -190,8 +189,7 @@ Task specific parameters for different tasks (text generation/image generation/e | option | Value format | Description | |---------------------------|--------------|--------------------------------------------------------------------------------| | `--num_streams` | `integer` | The number of parallel execution streams to use for the model. Use at least 2 on 2 socket CPU systems. Default: 1. | -| `--model_type` | `string` | Type of the source TTS model: `speecht5` (default) or `kokoro`. | -| `--vocoder` | `string` | The vocoder model to use for text2speech. For example `microsoft/speecht5_hifigan`. | +| `--model_type` | `string` | Type of the source TTS model. Supported values: `kokoro` (default, recommended) and `speecht5` (legacy). | ### Speech to text | option | Value format | Description |